Pre-testing Sensitive Questions: Perceived Sensitivity, Comprehension, and Order Effects of Questions about Income and Weight

Several new questions dealing with eating habits, family income, eligibility for food stamps, and body mass index are being considered as a supplement to the American Time Use Survey. This paper discusses methods used for pretesting sensitive questions and addresses concerns about possible order effects of the sensitive questions. New questions were pre-tested with two phases of cognitive interviews designed to assess comprehension, perceived sensitivity of questions, and order effects. Special attention was paid to the questions about income and weight as they were anticipated to be the most sensitive questions proposed for the supplement. Cognitive testing results suggested that the income question was difficult for participants to respond to and was perceived to be the most sensitive question in the supplement, followed by weight. The second phase of testing randomly varied the order of income and weight questions to assess order effects in perceived sensitivity. Income was always reported to be more sensitive than weight, whether it came before or after weight. Yet, the weight question generated item nonresponse and more visual signs of discomfort from participants. Cognitive testing methods for new item development and results of testing are discussed. Introduction The American Time Use Survey (ATUS), which began production in 2003, is a household survey that uses a computer assisted telephone interview as the mode of data collection (see Horrigan & Herz, 2004 for a more detailed description). The interview gathers data from one designated person in a selected household about his or her time spent in daily activities during the previous 24hour period. Open-ended descriptions of activities, duration (or actual starting and stopping times), and questions about with whom and where the activity occurred are also collected in the form of a time diary. In addition, several summary questions immediately following the diary ask about work, childcare, volunteer activities and trips away from home for two or more nights in a row. Recently, several new questions that are designed to measure eating habits, weight, and income have been proposed to be added to the interview in the form of a supplement. These questions were designed to investigate the relationship between these variables and how people use their time – for example, the possible relationship between income and weight and time spent in food preparation and shopping. A total of 11 questions (or possibly less, depending on skip patterns) have been proposed for the supplement. Before these questions could be added to the ATUS, the questions needed to be pretested in order to assess participants’ ability to comprehend and respond to them. Cognitive interviewing, a qualitative technique for assessing question quality, was chosen as the primary method for pre-testing these questions. Cognitive interviewing is a technique that studies how participants understand, mentally process, and respond to questions being asked in the survey (Willis, 2005). The goals of testing the supplement questions were to find out more about whether or not participants comprehend the questions as they were intended and how easy or difficult it was to process and respond to the questions asked (i.e., is it possible for people to respond in a consistent and coherent way that will produce quality data). In addition to these goals, the cognitive testing of the supplement questions had a third goal. Specifically, two of the questions proposed in the supplement were identified as potentially sensitive in nature, and the cognitive testing was designed to assess whether these questions would cause an adverse reaction in participants and potentially generate item nonresponse. The two questions that were identified as potentially sensitive were questions about weight and income. Previous research has shown that questions about income are among the most sensitive questions that can be asked in a survey. Laumann, Gagnon, Michael, & Michaels (1994) found that income questions were perceived to be even more personal and sensitive than some questions about sexual behavior. Therefore, it was predicted that the proposed income question might be seen as very sensitive and cause participants discomfort. In addition, a question about how much a respondent weighs was predicted to generate similar reactions. In addition to weight and income potentially being perceived as sensitive questions, it was anticipated that comprehension, and, thus, ability to respond accurately to the income question might be difficult. The income question initially proposed for the supplement was a highly detailed question that included a list of items to keep in mind and gave a time frame (see Methods Phase 1 section for exact question wording). Long lists of items to include or not include are difficult for participants to keep in mind when answering a survey question (Fowler, 1995). In addition, the specific income questions that government agencies typically prefer have been shown to have comprehension and data quality problems (Moore, Stinson, & Welniak, 2000). Therefore, it was anticipated that the wording of the income question would be difficult to comprehend, and that the question would be difficult to answer correctly. This paper focuses on the cognitive testing results of the two questions that were identified as potentially sensitive: weight and income. Two phases of testing were undertaken that assessed comprehension, response quality, and perceived sensitivity of the weight and income questions. The first phase of testing took a primarily qualitative approach. In the second phase of testing, a more quantitative approach was taken in order to assess whether respondent gender or the order of the sensitive questions were related to the perceived sensitivity of the questions. Results from both phases will be discussed.